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APOLLO DOliAIN AUCHITECTURE 



1.1 ARCHITECTURE EVOLUTION: 
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The Apollo DOMAIN system has evolved as a direct result of 
improvements in technology and is widely held to be the 
architecture of the 1900's. It combines the good parts of both 
timesharing and dedicated minicomputers, but eliminates the 
disadvantages of both of these earlier forms. The Apollo DOUAIN 
system lias good sharing capabilities provided by a high speed 
interactive network as well as interactiveness provided by a 
dedicated computer available to each user. 
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1.2 G0VKIMIII1G PRINCIPLES: 

There are several principles that have been used to Govern the 
design of the Apollo computer system. First, and foremost, is 
the notion that there exists a dedicated CPU for each user. 
Second, each user is interconnected with a high performance 
local area network. Third, the design of the architecture is 
besed on hi^n level abstractions so that we may independently 
evolve lower level components (such as the instruction set, or 
internal buses) with minimum impact. Fourth, is the use of 
advanced technologies, such as VLSI, Winchester disks, and so 
on . 
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1.3 HIGH LEVEL IMPLEMENTATION: 

The Apollo system incorporates designs which are uniformly 
advanced, or appear at a higher level than conventional 
computers. A conventional computer is characterized by: (1) a 
machine level instruction set or what we call an ISP, (2) a 
Machine level address space or a virtual address space which is 
a Measure of the range of addressing that the computer can span, 
(3) the processor memory bus organization, or what we call PMS, 
including the memory buses, the attachment of processors, the 
cttachment of multiple memory units and so on, and ( H ) the I/O 
system of the computer, or the I/O bus. 
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i.'i advauci;d concepts: 

There are iiu.riy advanced concepts that have been applied to the 
Apollo architecture and they can be roughly broken down into 
three general categories: (1) those pertaining to the overall 
system environment, (2) those pertaining to the program 
environment, (3) those pertaining to the user environment. It 
is useful to point out certain particular features that have 
been incorporated into the DOMAIN system in each of these 
environments. 

The Apollo system environment is unique in the sense that: (1) 
The architecture is based on a network as opposed to a central 
systems architecture, (2) a network which allows shared data and 
peripherals, (3) a network oriented object based operating 
system that will be described in more detail later. 

The processing environment for the Apollo system includes: (1) 
a very large linear address space for virtual memory 
management, (2) advanced concepts, such as stream I/O which 
will be described later, (3) new ideas such as shell 
programming which allow people to build procedures at the 
command level. 

The user environment of the Apollo DOMAIN system is radically 
different from conventional systems. Rather than a character 
oriented dumb terminal, the Apollo system has for each user an 
integral bit map display. This parallel device allows many 
concurrent programs to be executing on behalf of each individual 
user, which is accomplished by dividing the screen into multiple 
independent window areas. 
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II. 1 SYSTEM CliVIKONMEHT OBJECTIVES; 



network modularity is a principal design objective of the Apollo 
computer system, providing a wide range in performance, a wide 
r*c rif.e in growth capability, and a wide range in system level 
availability. Modularity at the network level allows users to 
incrementally expand their system by themselves on their site, 
and without substantial programming. It means that they can 
replicate nodes to obtain very high availability. It further 
!i'eari3 that the overall system configuration can conform to the 
users specific application in the most cost effective way he 
chooses. From a manufacturer's point of view, network 
nodularity significantly eases system maintenance, allowing the 
replacement of entire nodes as well as the ability for one node 
to diagnose another. 

A second design objective for the Apollo system environment was 
to incorporate a high performance coaxial local area network. 
Although our system is designed to accommodate any two address 
packet transport mechanism, the specific implementation that 
Apollo has chosen involves a ring topology. Rings have numerous 
advantages over alternative approaches: They generally allow 
higher data bandwidths and longer distances, they allow 
migration to new technologies such as fiber optics, they are 
very interactive allowing very fast network arbitration, and 
finally they incorporate a free acknowledgement function with 
the circulation of each packet. 

A third system environment objective v/as to maximize network 
interactiveness. In tills regard, our design eliminates all 
superfluous message buffering between nodes, allowing a message 
generated from one process to be transmitted directly to another 
process on a separate machine. Secondly, our network controller 
transmits data through the block multiplexor channel which 
allows all high performance DMA devices to have acces3 to the 
total memory bandwidth of both machines. Consequently, when a 
message is transmitting from one machine to another, the data 
rate is at the maximum possible permitted by the two memory 
systems . 
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.11.2 SYSTEM ORGANIZATION: 

The system level organization of the Apollo system is based on 
the Apollo DOMAIN network. This network allows an extremely 
wide range in performance , growth and system availability. 
Moreover, users attached to the system can intercommunicate, can 
cecess shared programs and data files across the network, can 
i cccss common pools of peripherals, and can finally access 
remote facilities, including larpe foreign machines or other 
Apollo DOMAIN systems. Consequently, the Apollo DOMAIN network 
tor, ether with the per U3er computing node is intended to provide 
an entire computing facility to each user. 
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II. 3 HIIIG UETWOIIK PROTOCOL 

The Apollo DOMAIN systew is designed around a two addresQ packet 
transport network. The specific implementation of this network 
can take various forms, and the system is specifically designed 
to be able to nitrate from one form to another as the technology 
requires. 

The topology of the Apollo network is in the form of a circular 
ring. Access to this ring is arbitrated through the passing of 
a TOKEN which is a specific encoding of bits passed from one 
node on the network to another. The system allows one and only 
one TOFCEIJ to be on the ring at any given instant, and the 
possession of this single TOKEM gives a particular node 
exclusive use of the network. 

The format of the message on the ring includes the destination 
node address, the source node address, header information, data, 
a CMC check, and finally an acknowledgement field. The 
acknowledgement field id adjusted by the destination node, 
thereby acknowledging the correct receipt of the packet to the 
source node. 
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HIT SYr.TMII IlIKKAnCIIY: 



Tlio Apollo central processing unit is built around a VLSI 
Microprocessor with 32 bit architecture. The instruction set of 
the processor includes both 32 bit data types as well as a 32 
bit linear virtual address space. The physical parameters of 
the system, most notably the width of the data path, can be 
viewed in a lii ere rchical arrangement. At the system level 
computer nodes are interconnected with a 1 bit serial packet 
network. Certain peripherals attached to an individual computer 
node are interconnected with 8 bit (1 byte) data paths, whereas, 
the memory system and high performance peripherals operate on a 
16 bit data path. Internal CPU registers and arithmetic logic 
unit are all implemented with full 32 bit data paths. 

Consequently, the CPU is generally 32 bits wide, the memory 
system is generally 16 bits wide, while the network system is 
only a single bit wide. The width of the data path varies 
inversely with the physical distance from the internal 
processing registers. 
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II. 5 NODE OHGAIIIZATIOII: 
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Although the display memory and the program memory are in 
separate physical bus organizations, they actually share the 
3ame address space so that the CPU can instantaneously access 
display memory and alter its contents. Furthermore, the bit 
mover can move display areas (rectangles) Into and out of 
program memory. The system is designed so the CPU can access 
program memory and the display memory can refresh to the CRT 
display, and the bit mover can be moving rectangles all in 
parallel and without interference. 
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Finally, the display system is comprised of a separate 
autonomous 1/8 Megabyte bit map memory which is organized into a 
square array of 102*J bits on each side. The display memory is 
constantly refreshed onto an 800 x 102*J bit map CRT. There is a 
separate bit mover which is capable of moving rectangles from 
one part of the screen onto another part of the screen at a data 
rate of 32 megabits per second. 
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II. 6 BIT MAP DISPLAY: 



The bit nap display system is comprised of a 102*1 bit by 102*1 
bit array. A rectangular region of 600 by 102M is physically 
transferred onto the CRT display. The remaining area is used as 
temporary storage for character font tables. The bit mover is a 
hardware primitive which is capable of moving a rectangular area 
from any place on the screen to any other place on the screen. 
This primitive is used to move windows into and out of main 
Memory, to move them relative to the screen Itself, to implement 
scrolling and to create character strings from character fonts. 
The bit mover operates at 32 megabit per second data rate when 
moving entirely within the display memory. 

The bit mover can move bit aligned rectangles from display 
i.:ei.iory to/ from work alligned buffers in program memory where 
the CPU can efficiently perform raster operations, such as 
exclusive orinr; two or more graphic representations. 
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IJ.I.1 PHOCESSIIKS LIIVIHOHIIENT OBJECTIVES: 

A principal objective in designing a system processing 
euv ir omnent was to abstract common entities, like programs and 
data files, into a uniform abstraction which vie call an object. 
The totallity of objects across a network forms a 96 bit virtual 
address space which is comprised of two fields: a unique object 
nai.ie consisting of 6*1 bits, and a 32 bit byte address within an 
object. A second objective was to provide a demand paged 
operating system to implement a network wide virtual memory. A 
third objective was to provide an environment for efficient 
process to process streaming and the control of this streaming 
through shell programs. Finally, an efficient compiler, binding 
and execution procedure whereby network wide programs can be run 
interactively. 
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III. 2 SYSTK11 IIAMR SPACES: 



He now turn to the operating system design in the Apollo DOMAIN 


system. One way of viewing a complex system is to enumerate and 


describe the various name spaces that occur in the system. For 


example: First, there is the user global namespace, or what the 


user would normally type at a terminal to execute a program or 


access a data file. Second, there is the system global 


namespace, or the namespace that the operating system uses at a 


network level. Third, there is an object address space. Our 


object address space is 32 bits long and contains programs and 


files as well as other entities in the operating system which 


I'll describe later. Fourth, there is a process virtual address 


space that rcspresents an address space in which a Motorola 


GGOOO process executes. Fifth, there is the physical address 


space which represents the amount of physical memory that can be 


placed on the system. Sixth, there is the network address space 


or the maximum number of nodes that can be placed on the 


network. And, finally, there is the di3k address space or the 


maximum of , by tea or pages that disk can hold. 


In the Apollo system the user global namespace is syntactically 


represented as a stream of characters separated by slashes. 


This actually represents a hierarchical tree space which I will 


describe later. The system global namespace is a 96 bit address 


space comprised of a UID which is 6*1 bits and an offset which is 


32 bits wide. The 6*\ bit UID is unique in space and time. It 


is unique in space in that it includes an encoding of the 


machine's serial number and it is unique in time in the sense 


that it includes the time at which the name was created. Thi3 


guarantees that Tor all time in the future and for all machines 


that Apollo builds, no two machines will ever create the same 


UID, hence the term unique ID. 


UID' s are names of objects. Objects are used to hold programs, 


tiles and various other entities in the Apollo system. An 


object is a linear 32 bit address space, byte addressable, and 


can be located generally any place on the network. Objects are 


the primary focus for the Apollo DOMAIN system and are cached 


into the process address space provided by the Motorola 68000. 


This process address space, while very large, is still 


considerably smaller than the 32 bit object address space. 


Consequently, address regions of an object are mapped into 


regions of a process in much the same way that regions of 


physical memory are frequently mapped into regions of a cached 


memory . The process address space is a 2*» bit virtual address 


which is converted to a 22 bit physical address by memory 


management hardware. The unit oT allocation in the physical 


address space is 102*1 byte pages. 
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III. 3 SYSTEil RF.LATIOHSHIPS: 
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111,1 SYSTEM VIIITUAL ADDRESS SPACE: 
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111,6 PROTECTION/ STATISTICS: 

At each access to a page a set of rights (execute, read, write) 
are checked as a function of a particular level that the process 
is running at. The protection hardware specifies the particular 
rights at tills level and all higher levels. The levels are two 
supervisor levels and two user levels. 

The nci.iory canageiient hardware automatically records and 
Maintains certain statistics about the page access. In 
particular a bit is set every tine a page is accessed and a 
second bit is set when that page is modified. The operating 
kernel scans these bits periodically to' maintain knowledge of 
the statistical usage of the pages for the purpose of page 
replacement . 
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III. 7 I/O MAPPING: 

Peripherals on the MULTIBUS are mapped into the 22 bit Apollo 
physical address bus by means of an I/O map. The I/O nap 
consists of 256 page entries, each entry pointing to a 
particular Apollo page. A peripheral on the MULTIBUS can 
concrete a 16 bit word or byte address and have the high order 
bits indexed into the page map and the low order bits indexed 
relative to the page. In this way MULTIBUS peripherals can 
directly address themselves into the virtual memory of a 
process. 
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III. 9 DISK STRUCTURE: 

Objects are mapped onto physical disk3 using a rather dynamic 
ctorane allocation. First of all a disk structure contains a 
physical volume label which is* a list of pointers which point to 
multiple logical volume labels. The division of a physical 
volume into multiple logical volumes is a means whereby fixed 
partitions can be created which do not compete for common 
storage. In other words, one can create a logical volume and 
. guarantee it hts a certain minimum amount of allocation. 



C a c h logical 


volume label contains a 


volume table 


of 


contents 


map. The volume table 


of conten 


ts is a list 


of all of the 


object UID's 


in that volume 


and for each object a set of 


object 


retributes. 


The object 


attributes 


consist of the 


ob je 


ct type, 


access contro 


1 information, 


accounti 


ng information 


(la 


s t date 


accessed, la 


st date modified), and 


a map to all o 


f the 


various 


d a t a blocks w 


hi ch comprise 


the object 


The map is 


compr 


ised of 


3 5 pointers. 


The first 


32 point 


ers point directly 


to data 


blocks each of which cons 


ists of 


a single page 


The 3 3rd 


pointer poin 


ts to a block 


of second 


level pointers 


(256 


of them) 


which in turn 


point to actual data 


blocks. The 


3*»th 


pointer 


expands into 


three level 


s of storage and the 


35th 


pointer 


expands into 


four levels of 


storage. 


Consequently 


, fo 


r small 


objects data 


access is 


very effi 


cient; and for 


large 


objects 


.'Horace allocation is very 


efficient. 









Siacli block contains not only 102*1 bytes of data, but also the 
DID and object page number that this page represents. 
Consequently if a failure should occur, the entire mapping 
structure can be recreated by a single pass over all of the data 

pages. 
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III. 10 I/O HIERARCHY: 
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III. 11 STREAM I/O: 

The stream I/O level deals with the Interconnection of objects, 
including process to file operations, and process to process 
operations. It has the principal characteristic of being object 
type independent. And since it is implemented through the 
napped I/O level, objects can be conceptually interconnected by 
streams both within the same node and across the network. 

When streams are used to interconnect processes, the output of 
one process is connected to the input of another process. This 
multiple process application can acquire the form of a stream 
filter whereby every process forms some transformation on its 
input and then passes the output to another process. When 
applications are encoded in this manner, programmers are 
encouraged to write processes as simple, modular programs that 
perform some primitive function.* Frequently, these functions 
can be reused across many applications. 
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III. 12 SOFTWAHF.; TOOU 



A largo collection of program Modules designed to perform sone 
primitive function have evolved over years of use by a large 
collection of users. These modules are referred to as Software 
Tools cud are widely distributed throughout the user community. 
Software Tools follows the nethodoloj-y laid out in the book 
entitled "Software Tools" by Kernigan and Plauger, published by 
Addison Uesley. 
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Applications can be easily formed by interconnecting streams of 
data through r collection of Software Tools. The collection of 
standard Software Tools is derived from a library of programs - 
a "toolbox" of Software Tools. In this way complex applications 
can frequently be formed with little or no programming. The 
tii.:e required to develop a new application i3 significantly 
reduced. Furthermore, users are encouraged to write programs 
that are small, conceptually simple, and usable for many 
applications and by many users. 
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III. 13 SIIKLL PHGGHAHS: 



A shell program Is a higher level flow of control above the 
conventional program level (e.g. Fortran or Pascal). Shell 
programs are written in a shell programming language that has a 
rich set of constructs that are, in many respects, similar to a 
conventional language. However, an executable statement within 
a shell program frequently involves the complete execution of 
one or more conventional programs. In thi3 regard, a shell 
program can be thought of as a sophisticated command processor 
which coordinates the execution of multiple program steps. 

The ability of users to program applications . in a shell 
programming language relieves a great deal of complexity that 
would otherwise be required within a Fortran or Pascal program. 
Consequently, programs written in these languages tend to be 
simpler and have fewer input options. 

The concept of shell programming goes hand - in-hand with the 
concept of Software Tools. Here, the shell programs represent 
the interconnect of streams between various programs, and can be 
extended to richly interconnect small programs in order to form 
complex applications. 
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execution if there are no 


unresolved references (i.e., 


no 


other 


subroutines which need 


to be 


bound together). If 


the 


i\ p p 1 1 c a 


tion contained 


several 


source 


program 


objects, these 


cowpile 


d objects must 


be bound togetr, 


er prior to execution, a 


process 


accomplished by 


the DINDER. The 


process 


of loading 


and 


oxocuti 


ng a compiled 


object consists 


of: (1) 


Mapping the 


pure 


position independent code into a 


region 


of a 


process add 


ress 


space. 


(2) Creating an impure d 


ata object and 


mapping that 


data 


object 


into an inpure section of 


the process add 


ress space. 


(3) 


Dynamically linking operating sj 


3 1 e m r e f 


erences 


to the opera 


ting 


Gyste t.i 


during execution. 













There are two important points in this procedure: (1) The 
output of a compiler can be directly executed if there are no 
external references to be resolved. (2) A compiled object, once 
formed, is never referenced again until it is in execution. 
This represents a very efficient compile and run time design* 
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III. 15 C0I1PILKD OnjIICT: 

The compiled object forraat is comprised of two parts: The first 
Major part is position independent code and pure data which is 
directly mapped and executed into a process address space. The 
second part is a database used by the loader to create an impure 
temporary data object which is subsequently mapped into the 
impure part of a process address space. 
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IV. 1 UoCH EHVIHUMiKIIT OBJECTIVES: 

A key objective in designing the Apollo user environment in to 
conbine simplicity and uniformity with a high degree of 
functionality. 



All objects that the system is capable of referencing can be 
expressed in a uniform name space that transcends the entire 
network. further, a bit map display, as opposed to a character 
display, is used to represent text and graphics output. The 
output frosi multiple programs can be concurrently displayed 
through multiple windows, thereby providing a degree of 
functionality unavailable on conventional systems. 
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IV. 3 COUCH IIRRIIT USER Elf VI llOllf IEIIT 



The no 


tion of concurrency is a 


new concept on the 


Apollo DOMAIN 


systc in 


unavailable on convent 


ional timesharing 


systems. 


On 


these 


latter systems users are 


generally required t 


o execute 


one 


function at a time. When a U3er 


switches from one 


function 


to 


ano tli er 


, generally the context 


of the previous function is 1 


ost 


a n d ha s 


to be subsequently recreated. The Apollo 


integral 


bit 


i.i a p d i 


splay provides the user 


with the capability 


of display 


ing 


MUltipl 


e windows simul taneously . 


Each window can 


contain 


the 


output 


of related or unr elate 


d applications. For 


example, 


one 


window 


can contain the sequentia 


1 output of a proG 


ram while 


a 


second 


window graphically displ 


ays the accumulated 


output of 


the 


y a n e p 


rocran. Similarly, pro 


gram development, 


compilati 


on , 


editing 


and an on-line help 


system can all be 


concurrer 


tiy 


display 


ed. 









Consequently, the Apollo system is designed to accommodate a 
total user environment, which we believe always involves a 
number of concurrent functions. 
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he screen manager represents the outer most layer of logic 
ithin the Apollo system -- that which controls the rela t ionship 
none the Many windows projected onto the CRT screen, 
ccordingly, the Apollo system adds two additional layers above 
he conventional programming level. As mentioned earlier, a 
rogrammabl e shell coordinates the activity of many programs (in 
oth parallel and sequential relationships). The output of this 
hell is written into a virtual terminal, called a PAD. 
options of this PAD are displayed through a rectangular window 
hi cli is then projected onto the CUT display. 



The screen Manager permits multiple windows to be displayed 
concurrently, each of which can be executing an independent 
shall or command environment. The philosophy of the screen 
manager is to allow programs to output data in a logical format, 
while allowing the user to independently control what is 
physically displayed. 

The screen manager is controlled by the use of function keys on 
the user keyboard. Pushing a function key causes the execution 
(interpretation) of a user programmable sequence of screen 
manager primitives. Consequently, the user can define function 
keys to perform complex screen manager functions. 
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The A i 


olio DONA I II operating system creates a degree of 


indepeiK 


once between application programs and what is actually 


v iewed 


on the terminal screen. In particular, application 


programs 


create virtual terminals which we call pads. The pads 


a re in d 


ependently windowed onto the CRT screen totally under 


control 


of the user. Window images are superimposed on the pads 


and cen 


be moved relative to the pad in either a horizontal or a 


vertical 


direction. V/indow images from various pads are stacked 


logical! 


y on top of the screen so that only the one on top is 


display g 


d. Consequently the user environment i3 actually a 


three dimensional volume: 800 bits going across, 102*1 bits going 


down and 


many levels of windows deep. The user can also move 


window 


areas up or down relative to the physical screen and 


finally 


can move window areas into and out of the screen 


relative 


to other window areas. 


Programs 


create the pad by writing command and data sequences 


through 


a stream. The window image created by the screen 


Manager 


from the pad can be placed anywhere in the CRT and can 


be ovcrl 


ayed by other window images. V/indow images contain 


lines 


and frames. A line is a single line sequence of 


characters and has only one dimension. A frame has two 


dimensions and has a rectangular format. It contains characters 


and /or g 


raphic data. Finally, frames may also contain user 


created 


bit maps. These bit maps may reside either within the 


pad or w 


ithin a separate user supplied object. Pad information 


normally 


accumulates over the life— -o-f a process. This allows a 


user to 


scroll cither in reverse or in forward directions over 


the entire life of the process. However, for efficiency sake 


certain 


commands may be emitted from the program to delete all 


or part 


of the pad as appropriate. 



p£ucesjfe > 




(vwtoav. Tect*iuft.s 



1/OiUDOvos 



Prel in in cry- 



January .72, 1 9H 1 



V.I SUIIIIAPY OF KEY POINTS 

An Apollo computer system is comprised of a number of high 
pcrforiaance dedicated computers interconnected over a local area 
network. I^ach of these nodes contains a large machine 
architecture which implements a demand paged network wide 
virtual memory oystem, allowing a large number of processes for 
each user, each process having a very large linear virtual 
address space. Languages that .run on the Apollo system include 
Fortran 77 and Pascal and are implemented to take advantage of 
the machine's 32 bit orientation. 

An object oriented network operating system coordinates the 
user's access to network wide facilities. Objects themselves, 
representing programs and data file3, etc., are independent of 
their network location, and given appropriate access rights, can 
be accessed uniformly by anyone on the system. 

The user's display terminal is capable of displaying multi-font 
text, graphics and can be divided into multiple windows each 
displaying independent program output. 

The Apollo system is designed around high technology. It 
incorporates VLSI CPU chips, large capacity Winchester disk, and 
advanced communication technologies. 
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